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9.1.7 Estimation for Random Vectors 

The examples that we have seen so far involved only two random variables X and Y. In practice, 
we often need to estimate several random variables and we might observe several random variables. 
In other words, we might want to estimate the value of an unobserved random vector X: 
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Almost everything that we have discussed can be extended to the case of random vectors. For 
example, to find the MMSE estimate of X given Y = y, we can write 


£[.x 1 |y 1 ,y 2 ,...,y„] 
e\x 2 \y 1 } y 2 , ■■■,¥„] 


±„ = £[X|Y] = 


LE[x m |y 1 ,y 2 ,...,y„]J 


However, the above conditional expectations might be too complicated computationally. Therefore, 
for random vectors, it is very common to consider simpler estimators such as the linear MMSE. Let's 
now discuss linear MMSE for random vectors. 


Linear MMSE for Random Vectors: 

Suppose that we would like to have an estimator for the random vector X in the form of 
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Xi — AY + b, 


where A and b are fixed matrices to be determined. Remember that for two random variables X 
and Y, the linear MMSE estimator of X given Y is 




Cov(X, Y) 
Var(Y) 1 
Cov(X, Y) 
Cov(Y, Y ) 1 


EY) + EX 
EY) + EX. 


We can extend this result to the case of random vectors. More specifically, we can show that the 
linear MMSE estimator of the random vector X given the random vector Y is given by 

± L = C xy C y ~ 1 (Y-.E[Y]) + .e[X]. 

In the above equation, Cy is the covariance matrix of Y, defined as 

C Y = E[(Y - EY)(Y - EY) t ], 

and Cxy is the cross covariance matrix of X and Y, defined as 

Cxy = E[(X- EX)(Y - EY) t }. 


The above calculations can easily be done using MATLAB or other packages. However, it is 

sometimes easier to use the orthogonality principle to find X^. We now explain how to use the 
orthogonality principle to find linear MMSE estimators. 


Using the Orthogonality Principle to Find Linear MMSE Estimators for Random 
Vectors: 


Suppose that we are estimating a vector X: 
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given that we have observed the random vector Y. Let 
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x L = 
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be the vector estimate. We define the MSE as 


MSE = J2E[(X k -X k f 


k =1 


Therefore, to minimize the MSE, it suffices to minimize each E [(Xfc — X/ c ) ~ | individually. This 
means that we only need to discuss estimating a random variable X given that we have observed the 
random vector Y. Since we would like our estimator to be linear, we can write 

n 

X-L = + b. 

k= 1 

The error in our estimate X is then given by 

X = X - X L 

n 

— X ^ ^ b. 

k =1 


Similar to the proof of Theorem 9.1 . we can show that the linear MMSE should satisfy 

E[X] = 0, 

Cov(X, Yj) = E[XYj\ = 0, for aU j = 1,2, ■ • • ,n. 

The above equations are called the orthogonality principle. The orthogonality principle is often stated 

as follows: The error (X) must be orthogonal to the observations (Y\, Y2, • • % Y n )- Note that there 
are Tl + 1 unknowns (aj, a 2 , * • *, an d b) and Tl + 1 equations. Let us look at an example to 
see how we can apply the orthogonality principle. 


Example 9.9 

Let X be an unobserved random variable with EX = 0, Var(X) = 4. Assume that we have 
observed Y\ and Y 2 given by 
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Yi = X + W u 
Y 2 = X-hW 2 , 

where EW\ = EW 2 = 0, Var(Wi) = 1, and Var(W 2 ) = 4. Assume that W \, W 2 , 
and X are independent random variables. Find the linear MMSE estimator of X, given Y\ and Y 2 . 

• Solution 

o The linear MMSE of X given Y has the form 

X L = aY\ + bY 2 + c. 

We use the orthogonality principle. We have 

E[X\ = aEYi + bEY 2 + c 
= a- 0 + 6- 0 + c = c. 

Using E [X] = 0, we conclude C = 0. Next, we note 

Cov(X l ,Yi) = Co v(aY 1 + bY 2: Y x ) 

= aCov(Yi, Yi) + bCov{Y lt Y 2 ) 

= aCov(X + Wi,X + W\) + 6Cov(X + W\,X + W 2 ) 
= a(Var(X) + Var(W r i)) + 6Var(X) 

= 5 a + 46. 

Similarly, we find 

Cov(X L , Y 2 ) = Cov(aFi + &Y 2 , Y 2 ) 

= aVar(X) + 6(Var(X) + Var(W 2 )) 

= 4a + 86. 

We need to have 

Cov(X, Yj) = 0, for j = 1,2, 

which is equivalent to 

Co v(X L ,Yj) = Cov(X, Yj), for j = 1,2. 

Since Cov(X,Yi) = Cov(V, Y.) = Var(X) = 4, we conclude 

5a + 46 = 4, 

4a + 86 = 4. 
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Solving for a and b, we obtain a — |-, and b = 
estimator of X, given Y\ and Y 2 , is 




2 1 
3 Fi+ 6 


previous 
next —> 
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